Phylogenetic analysis in the anomaly zone.

The concatenation method has been widely used as a means of combining data to estimate phylogenetic trees (Huelsenbeck et al. 1996a, 1996b; Glazko and Nei 2003). However, simulation studies have shown that the maximum likelihood (ML) estimate of the species tree for concatenated sequences may be statistically inconsistent if the gene trees are highly heterogeneous (Kolaczkowski and Thornton 2004; Kubatko and Degnan 2007). Recently, Degnan and Rosenberg (2006) defined an “anomaly zone”—a set of short internal branches in species trees that will generate gene trees that are discordantwith the species treemore often than gene trees that are concordant. Kubatko and Degnan (2007) went on to show that when DNA sequences are generated from gene trees simulated from species trees in the anomaly zone, as well as from species trees slightly outside this zone but still with short internal branches, the ML estimate of the species tree for the concatenated sequences can be inconsistent, resulting in increasing certainty in thewrong species tree. These studies were all performed with a molecular clock on rooted gene and species trees within the variation realized in stochastic simulations of DNA sequences under the Jukes and Cantor (1969) model of nucleotide substitution. They applied the ML method with a clock to recover phylogenetic trees from their simulated concatenated data sets. In this paper, we show that phylogeneticmethods that solely utilize the relative order of divergences among a set of DNA sequences as a criterion for inferring phylogenies, such as the unweighted pair group method with arithmetic mean (UPGMA), are statistically consistent even when DNA sequences are generated from gene trees simulated from species trees in the anomaly zone. In addition, we use simulation to assess the performance of a variety of tree constructionmethodswhen analyzing concatenated sequences generated from 4 and 5-taxon species tree located in the anomaly zone and show that a variety ofmethods do in fact recover the correct species tree topology, whereas ML, with or without a molecular clock, remains inconsistent. However, the branch lengths of the tree inferred from concatenated data are inevitably overestimated, as predicted by theory. Finally, simulations also suggest that a newly proposed Bayesian approach for estimating species trees from multiple unlinked loci, BEST (Liu and Pearl 2007; Liu et al. 2008), is consistent in both topology and branch lengths on data sets generated from species trees in the anomaly zone. THEORETICAL RESULTS